Open qianmoQ opened 3 years ago
Supporting native Hive UDFs in Presto is not very straightforward as the UDF model is completely different. For example, Hive uses an initialize() function in GenericUDF to define type validation and inference. On the other hand, Presto uses type signatures. Since initialize() functions (used in Hive) are more expressive than type-signature-style type derivation system (used in Presto), it is hard to express the Hive UDF as a Presto UDF.
That said, at LinkedIn we use Transport to define portable user defined functions between Hive, Spark, and Presto. Once a UDF is written in this API, its equivalent Hive, Spark, and Presto versions are generated. Coral supports converting a Hive UDF written in Transport to its Presto equivalent transparently. Please see more details in this blogpost.
Supporting native Hive UDFs in Presto is not very straightforward as the UDF model is completely different. For example, Hive uses an initialize() function in GenericUDF to define type validation and inference.
Fortunately, we do not have to go as far. I think the goal is not to provide adapter interface allowing Hive UDF execution within Presto. Instead, it's to provide all and only the necessary conversions for view text. The input would be HQL view text (String) and the output is SQL view text, with Hive UDF function calls replaced with Presto function calls (or expressions, in case of no direct 1-1 function mapping).
This should be accessible to users via configuration or extension point.
Coral supports converting a Hive UDF written in Transport to its Presto equivalent transparently. Please see more details in this blogpost.
Using example provide by @qianmoQ -- let's say i implemented smid_udf
using Transport.
What would be the next step to have it recognized by Coral (as integrated in Presto)?
@findepi I also implemented smid_udf
function in Presto, but Coral couldn't parse it
Right now the mechanism is adding the mapping to Coral code. The current implementation should be easily evolvable to automatically map Transport UDFs since they follow a known pattern, and the inputs and outputs are the same.
If we want flexibility beyond changing the code, and beyond using Transport, I think a SQL API is needed at the Presto level to supply the mapping on the fly.
If we want flexibility beyond changing the code
we definitely cannot expect users to fork Coral and replace the version bundled with Presto with their own build. That would open all kinds of compatibility problems.
and beyond using Transport
@qianmoQ did you happen to implement your UDFs with Transport?
I think a SQL API is needed at the Presto level to supply the mapping on the fly.
Until functions are defined dynamically via SQL, we should not need to go that far. We could however need to provide a way to express those mappings for functions provided by custom plugins. Either the plugin could declare something, or there could be a mapping provided in the configuration somewhere.
cc @electrum
If we want flexibility beyond changing the code
we definitely cannot expect users to fork Coral and replace the version bundled with Presto with their own build. That would open all kinds of compatibility problems.
and beyond using Transport
@qianmoQ did you happen to implement your UDFs with Transport?
I think a SQL API is needed at the Presto level to supply the mapping on the fly.
Until functions are defined dynamically via SQL, we should not need to go that far. We could however need to provide a way to express those mappings for functions provided by custom plugins. Either the plugin could declare something, or there could be a mapping provided in the configuration somewhere.
cc @electrum
Did you say Presto UDF or Hive UDF?
Did you say Presto UDF or Hive UDF?
Probably Presto, but i do not understand yet how an UDF implemented with Transport is recognized by Coral. I think it should not matter how you implemented your Hive UDF, since it's not loaded in Presto at runtime at all.
@wmoustafa can you clarify how Coral recognizes UDF-s implemented with Transport?
@findepi
I implemented my custom UDF in Presto via the Implements Plugin,The source code is as follows
@SqlNullable
@ScalarFunction(value = "hello_scalar")
@Description(value = "print hello scalar")
@SqlType(StandardTypes.VARCHAR)
public static Slice smid_udf(@SqlType(StandardTypes.VARCHAR) Slice string) {
...
}
import com.facebook.presto.spi.Plugin;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Lists;
import com.google.common.collect.Sets;
import java.io.FileInputStream;
import java.io.IOException;
import java.net.URL;
import java.util.List;
import java.util.Set;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;
public class FunctionLoadPlugin implements Plugin {
private List<Class<?>> getFunctionClasses() throws IOException {
List<Class<?>> classes = Lists.newArrayList();
String classResource = this.getClass().getName().replace(".", "/") + ".class";
String jarURLFile = Thread.currentThread().getContextClassLoader().getResource(classResource).getFile();
int jarEnd = jarURLFile.indexOf('!');
String jarLocation = jarURLFile.substring(0, jarEnd); // This is in URL format, convert once more to get actual file location
jarLocation = new URL(jarLocation).getFile();
ZipInputStream zip = new ZipInputStream(new FileInputStream(jarLocation));
for (ZipEntry entry = zip.getNextEntry(); entry != null; entry = zip.getNextEntry()) {
if (entry.getName().endsWith(".class") && !entry.isDirectory()) {
String className = entry.getName().replace("/", "."); // This still has .class at the end
className = className.substring(0, className.length() - 6); // remvove .class from end
try {
classes.add(Class.forName(className));
} catch (ClassNotFoundException e) {
}
}
}
return classes;
}
@Override
public Set<Class<?>> getFunctions() {
try {
List<Class<?>> classes = getFunctionClasses();
Set<Class<?>> set = Sets.newHashSet();
for (Class<?> clazz : classes) {
if (clazz.getName().startsWith("io.edurt.tutorial.presto.scalar.function")) {
set.add(clazz);
}
}
return ImmutableSet.<Class<?>>builder().addAll(set).build();
} catch (IOException e) {
return ImmutableSet.of();
}
}
}
@wmoustafa can you clarify how Coral recognizes UDF-s implemented with Transport?
Good question, @findepi. Right now, it does not. However, it has a way to do that for Hive Generic UDFs. See this class for how this is implemented. The same approach is easily extensible to Transport UDFs, with the additional advantage that UDFs are easily mappable to their Presto versions in the case of Transport due to the straightforward mapping.
@findepi @wmoustafa What is the current progress?
@qianmoQ i am not aware of anyone working on this.
i'm Integrate Coral with Presto to enable querying hive views has error:
smid_udf
is a custom hive udf